• Implement cross-session judge-facts memory for eval command
  - Fetch retained facts per category at eval startup
  - Inject facts into judge prompts as "known patterns" section
  - Parse optional retain_fact from judge responses
  - POST novel facts back to server for future runs
• Add FormatKnownPatterns and ExtractRetainFact helper methods
  - Format facts as bulleted list with empty-state marker
  - Tolerant parsing of retain_fact field (handles code fences, null, empty, malformed JSON)
• Extend prompt template with "Known patterns" section and retain_fact response field
• Add 10 new unit tests covering fact formatting and extraction edge cases
• Fix 4 pre-existing TUnitAssertions0015 warnings in SetupCommandTests

Implement judge-facts fetching, formatting, extraction, and posting

• Added FetchAllJudgeFactsAsync to load retained facts per category at eval startup with
 per-category error handling
• Added FormatKnownPatterns to render facts as bulleted list with "(no patterns retained yet)" for
 empty state
• Added ExtractRetainFact to parse optional retain_fact field from judge responses, tolerating
 code fences and malformed JSON
• Added PostJudgeFactAsync to persist novel facts back to server after each judge invocation
• Updated BuildQuestionPrompt signature to accept knownPatterns parameter and inject into
 template
• Added Categories constant array for the four judge categories

src/kapacitor/Commands/EvalCommand.cs

Add JudgeFact and JudgeFactPayload models

• Added JudgeFactPayload record for writing facts to server (category, fact, source session/run
 IDs)
• Added JudgeFact record for reading facts from server (includes retained_at timestamp)
• Registered both types in KapacitorJsonContext for source-gen JSON serialization

src/kapacitor/Models.cs

Add tests for fact formatting and extraction

• Updated BuildQuestionPrompt_substitutes_all_placeholders test to include {KNOWN_PATTERNS}
 placeholder
• Added FormatKnownPatterns_returns_explicit_empty_marker_when_no_facts test
• Added FormatKnownPatterns_renders_bulleted_list test with multiple facts
• Added 8 ExtractRetainFact tests covering: present value, code-fenced response, absent field,
 explicit null, empty string, whitespace, non-string value, malformed JSON

test/kapacitor.Tests.Unit/EvalCommandTests.cs

Description

HandleEval only persists retain_fact after ParseVerdict succeeds, so any judge response that
includes a usable retain_fact but fails verdict parsing is silently not retained. This contradicts
ExtractRetainFact’s intent that retention plumbing “doesn't depend on verdict parsing succeeding.”

Code

src/kapacitor/Commands/EvalCommand.cs[R140-143]

+            // If the judge emitted a retain_fact, persist it for future evals.
+            if (ExtractRetainFact(result.Result) is { } retainedFact) {
+                await PostJudgeFactAsync(httpClient, baseUrl, q.Category, retainedFact, context.SessionId, evalRunId);
+            }

Evidence

The eval loop continues early when ParseVerdict returns null, so
ExtractRetainFact/PostJudgeFactAsync is never executed for that response. This is at odds with the
ExtractRetainFact doc comment stating retention shouldn't depend on verdict parsing success.

src/kapacitor/Commands/EvalCommand.cs[129-144]
src/kapacitor/Commands/EvalCommand.cs[216-245]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`retain_fact` persistence is currently gated on `ParseVerdict` succeeding. If the judge returns a response where `retain_fact` is present/valid but the verdict JSON is unparseable or fails validation (e.g., missing/out-of-range score), the loop `continue`s before calling `ExtractRetainFact`, so the fact is never retained.

### Issue Context
`ExtractRetainFact` is documented as independent of `ParseVerdict` so retention shouldn't depend on verdict parsing succeeding.

### Fix
Refactor the per-question loop to extract `retain_fact` regardless of verdict parse outcome, and only gate `verdicts.Add(...)` on `ParseVerdict`.

A typical structure:
- `var retainedFact = ExtractRetainFact(result.Result);`
- `var verdict = ParseVerdict(...);`
- if verdict != null -> add
- if retainedFact != null -> post

### Fix Focus Areas
- src/kapacitor/Commands/EvalCommand.cs[105-144]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Description

FetchAllJudgeFactsAsync only catches HttpRequestException; malformed/non-matching JSON from
/api/judge-facts can throw JsonException from JsonSerializer.Deserialize and crash eval startup.
This violates the comment that fact-fetch failures don’t abort the run.

Code

src/kapacitor/Commands/EvalCommand.cs[R251-265]

+            try {
+                using var resp = await httpClient.GetWithRetryAsync($"{baseUrl}/api/judge-facts?category={category}");
+                if (!resp.IsSuccessStatusCode) {
+                    Log($"Failed to fetch judge facts for {category}: HTTP {(int)resp.StatusCode}");
+
+                    continue;
+                }
+
+                var json = await resp.Content.ReadAsStringAsync();
+                var list = JsonSerializer.Deserialize(json, KapacitorJsonContext.Default.ListJudgeFact) ?? [];
+                result[category] = list;
+                Log($"Loaded {list.Count} retained facts for category {category}");
+            } catch (HttpRequestException ex) {
+                Log($"Could not load judge facts for {category}: {ex.Message}");
+            }

Evidence

FetchAllJudgeFactsAsync deserializes response JSON without catching JsonException, and HandleEval
calls it without a surrounding try/catch—so a JSON parse failure will propagate and abort the entire
eval. Elsewhere in the same file, JSON parsing failures are expected and handled via `catch
(JsonException)` (ParseVerdict).

src/kapacitor/Commands/EvalCommand.cs[93-99]
src/kapacitor/Commands/EvalCommand.cs[247-269]
src/kapacitor/Commands/EvalCommand.cs[312-320]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`FetchAllJudgeFactsAsync` can throw `JsonException` when `/api/judge-facts` returns malformed JSON (or an unexpected shape). The method currently only catches `HttpRequestException`, so this exception will bubble up and abort `HandleEval` before any questions run.

### Issue Context
The comment in `HandleEval` says failures fetching retained facts should not abort the run.

### Fix
In `FetchAllJudgeFactsAsync`, extend error handling to catch `JsonException` (and optionally `NotSupportedException`) around `JsonSerializer.Deserialize`, log an informative message, and `continue` to the next category.

### Fix Focus Areas
- src/kapacitor/Commands/EvalCommand.cs[93-99]
- src/kapacitor/Commands/EvalCommand.cs[247-269]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Description

ExtractRetainFact can throw InvalidOperationException when the response is valid JSON but not an
object (e.g., a JSON string/array), because it calls RootElement.TryGetProperty unconditionally and
only catches JsonException. This breaks the function’s tolerance guarantees and becomes more likely
if retain_fact extraction is moved earlier.

Code

src/kapacitor/Commands/EvalCommand.cs[R226-244]

+        try {
+            using var doc = JsonDocument.Parse(json);
+            if (!doc.RootElement.TryGetProperty("retain_fact", out var prop)) {
+                return null;
+            }
+
+            if (prop.ValueKind is JsonValueKind.Null or JsonValueKind.Undefined) {
+                return null;
+            }
+
+            if (prop.ValueKind != JsonValueKind.String) {
+                return null;
+            }
+
+            var text = prop.GetString()?.Trim();
+            return string.IsNullOrEmpty(text) ? null : text;
+        } catch (JsonException) {
+            return null;
+        }

Evidence

ExtractRetainFact unconditionally calls doc.RootElement.TryGetProperty(...) and does not check
ValueKind nor catch exceptions other than JsonException. TryGetProperty is only valid on object
elements, so valid non-object JSON can throw and crash the eval if this function is invoked on such
output.

src/kapacitor/Commands/EvalCommand.cs[223-245]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`ExtractRetainFact` assumes the parsed JSON root is an object and calls `TryGetProperty`. If the model returns valid JSON that isn't an object (e.g., `"oops"`, `[1,2]`, `true`), `TryGetProperty` can throw `InvalidOperationException`, which is not caught.

### Issue Context
The function is documented as tolerant; additionally, fixing the gating issue may cause `ExtractRetainFact` to run on more malformed/unexpected outputs.

### Fix
Add a guard:
- After parsing, if `doc.RootElement.ValueKind != JsonValueKind.Object`, return null.
Optionally broaden the catch to include `InvalidOperationException` as a belt-and-suspenders fallback.
Add a unit test for a valid-but-non-object JSON response (e.g., `"hi"` or `[1]`) returning null.

### Fix Focus Areas
- src/kapacitor/Commands/EvalCommand.cs[223-245]
- test/kapacitor.Tests.Unit/EvalCommandTests.cs[240-311]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Description

EvalCommand duplicates categories in both Questions[] and Categories[], so adding a new question
category later can silently skip judge-fact fetching/retention for that category. This creates a
latent drift bug.

Code

src/kapacitor/Commands/EvalCommand.cs[294]

+    static readonly string[] Categories = ["safety", "plan_adherence", "quality", "efficiency"];

Evidence

Questions define the categories that are actually evaluated, but FetchAllJudgeFactsAsync iterates
over a separate Categories array; if they diverge, known-pattern injection and retention will be
incomplete for the new category.

src/kapacitor/Commands/EvalCommand.cs[13-34]
src/kapacitor/Commands/EvalCommand.cs[247-266]
src/kapacitor/Commands/EvalCommand.cs[294-294]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
There are two sources of truth for categories (`Questions` and `Categories`). This can drift over time and silently break judge-facts fetching/retention for new categories.

### Fix
Derive `Categories` from `Questions` (e.g., distinct categories), or remove `Categories` entirely and iterate distinct categories from `Questions` within `FetchAllJudgeFactsAsync`.

### Fix Focus Areas
- src/kapacitor/Commands/EvalCommand.cs[13-34]
- src/kapacitor/Commands/EvalCommand.cs[247-266]
- src/kapacitor/Commands/EvalCommand.cs[294-294]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools